Skip to content

map_many: drop_none=True leaves out the results that are None - #16

Merged
lemire merged 3 commits into
mainfrom
drop-none
Nov 14, 2026
Merged

lemire merged 3 commits into
mainfrom
drop-none

Conversation

@lemire

@lemire lemire commented Nov 14, 2026

Copy link
Copy Markdown
Member

Builds on #14 (its first commit is #14's change; the diff against main includes it until #14 is merged).

map_many(fn, data, drop_none=True) leaves out the results of fn that are None, in the worker threads and on the one-thread path. When fn keeps only a few documents (a filter, a count), the threads no longer build a list entry for each of the others, and the calling thread no longer collects, walks and frees them.

Counting the stars of 7.6 GB of GH Archive events (fn returns the repository of a WatchEvent, else None: 90,403 results out of 2.2 million), big4, 3.14.10t, with #13, #14 and #15, end to end including the caller's Counter:

threads without drop_none=True
32 57.7 GB/s 68.2 GB/s
64 73.6 GB/s 88.0 GB/s
128 70.6 GB/s 88.4 GB/s

Tests: with and without threads, with fields, only None dropped (other falsy results kept), exceptions from fn and the json fallback unchanged. Full suite passes on 3.14.10 and 3.14.10t.

When fn keeps only a few documents (a filter, a count), the threads no
longer build a list entry for each of the others, and the calling thread
no longer collects and frees them. Counting the stars of 7.6 GB of GH
Archive events with fn returning the repository or None (90,403 results
out of 2.2 million) on big4, 64 threads, 3.14.10t: 73.6 to 88.0 GB/s end
to end, Counter included.
@lemire
lemire merged commit 76ce429 into main Nov 14, 2026
35 checks passed
lemire added a commit that referenced this pull request Nov 14, 2026
- Streams read read-only memoryviews of bytes and of read-only mmaps in
  place (#13).
- Parallel streams move their results into the list without reference
  counting (#14).
- fields= reads the documents with On-Demand (#15).
- map_many(..., drop_none=True) leaves out the results that are None (#16).
- Wheels for Pyodide (WebAssembly), Python 3.13 to 3.15; fixes for targets
  without SSE2 or NEON (dumps with ensure_ascii=False on strings of 16
  bytes or more) and for parse_many under Pyodide (#17).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant